Crate gzp

Parallel compression.

This module provides implementations of std::io::Write that are backed by an async threadpool that compresses blocks and writes to the underlying writer. This is very similar to how pigz works.

The supported encodings are:

• Gzip
• Zlib
• BGZF
• Mgzip
• Raw Deflate
• Snap Frame Encoding

References

• ParallelGzip
• pigz

Examples

A typical parallel compression task:

use std::{env, fs::File, io::Write};

use gzp::{deflate::Gzip, par::compress::{ParCompress, ParCompressBuilder}, ZWriter};

let mut writer = vec![];
let mut parz: ParCompress<Gzip> = ParCompressBuilder::new().from_writer(writer);
parz.write_all(b"This is a first test line\n").unwrap();
parz.write_all(b"This is a second test line\n").unwrap();
parz.finish().unwrap();

A typical single_threaded task:

use std::{env, fs::File, io::Write};

use gzp::{deflate::Gzip, syncz::SyncZBuilder, ZWriter};

let mut writer = vec![];
let mut parz = SyncZBuilder::<Gzip, _>::new().from_writer(writer);
parz.write_all(b"This is a first test line\n").unwrap();
parz.write_all(b"This is a second test line\n").unwrap();
parz.finish().unwrap();

If the number of threads might be 0, the following provides a uniform api:

use std::{env, fs::File, io::Write};

use gzp::{deflate::Gzip, ZBuilder, ZWriter};

let mut writer = vec![];
let mut parz = ZBuilder::<Gzip, _>::new()
    .num_threads(0)
    .from_writer(writer);
parz.write_all(b"This is a first test line\n").unwrap();
parz.write_all(b"This is a second test line\n").unwrap();
parz.finish().unwrap();

Re-exports

pub use crate::bgzf::BgzfSyncReader;

pub use crate::bgzf::BgzfSyncWriter;

pub use crate::mgzip::MgzipSyncReader;

pub use crate::mgzip::MgzipSyncWriter;

Modules

bgzf

Bgzf format base implementation.

check

Wrappers around different check types.

deflate

DEFLATE based compression formats.

mgzip

Mgzip format base implementation.

par

syncz

Single threaded compression that mimics the crate::par::compress::ParCompress API and implements crate::ZWriter.

Structs

Compression

When compressing data, the compression level can be specified by a value in this enum.

FooterValues

Pair

A Pair is used to represent header or footer information.

ZBuilder

Unified builder that returns a trait object

Enums

GzpError

Constants

BUFSIZE

128 KB default buffer size, same as pigz.

DICT_SIZE

32 KB default dictionary size, same as pigz.

Traits

BlockFormatSpec

FormatSpec

Defines how to write the header and footer for each format.

SyncWriter

Create a synchronous writer wrapping the input W type.

ZWriter

Trait that unifies sync and async writer

Type Definitions

CompressResult

Small helper type to encapsulate that the channel that sends to the writer is sending a receiver that will receive a result that is a tuple of the check value and the compressed bytes.